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Processes,  Models,  and 
Frameworks 


The  following  key  processes,  models,  and 
frameworks  are  synergistic  and  when  used 
together  provide  for  efficiency  and  effectiveness 
of  systems  engineering  and  software  acquisition 
and  development: 


Basic  Systems  Engineering 

I EEE/EI A  12207  Software  life  cycle  processes 

Capability  Maturity  Model  I  ntegration  (CMMI  ®)  Best 
Practices  framework 


Processes,  Models,  and 
Frameworks  cont. 


While  each  of  the  four  is  normally  taught 
separately,  the  establishment  of  a  software 
engineering  environment  consistent  with  all  four 
provides  for  an  infrastructure  that  can  be 
measured  for  effectiveness  and  improved  to 
provide  the  warfighter  with  better  quality 
products  faster  and  cheaper  - 

They  are  based  on  best  practices  throughout 
industry,  government,  and  academia  and  are 
designed  to  work  with  each  other 


Basic  Systems  Engineering 
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Systems  Engineering 


A  system  is  an  integrated  composite  of  people, 
products,  and  processes  that  provide  a  capability 
to  satisfy  a  stated  need  or  objective 

Systems  Engineering  (SE)  is  the  effective 
application  of  scientific  and  engineering  efforts 
to  transform  an  operational  need  into  a  defined 
system  configuration  through  the  top-down 
iterative  process  of  requirements  definition, 
functional  analysis  and  allocation,  synthesis, 
optimization,  design,  test,  and  evaluation 


Systems  Engineering  Process 


Define  customer  needs  and  required 
functionality 

Plan  the  project 

Document  the  requirements 

Validate  and  baseline  requirements 

Develop  the  design  based  on  the 
requirements 

Validate  that  design  meets  all  requirements  and  is 
baselined 

Produce  the  item  based  on  the  design 

Perform  system  validation 

Ensure  all  requirements  met  and  the  required 
functionality  performs  as  expected 


Design  Phase 


Coding/ 1  mplementation 


Testing 


Based  I  nternational  Council  on  Systems  Engineering  (I  NCOSE) 
definition 


Systems  Engineering  Technical 

Review  Process 


What  is  the  SETR? 

Systems  Engineering  Technical  Review  was  developed  by  NAVAI R  and  has  been 
adapted  for  use  within  the  entire  Navy 

An  iterative  program  timeline  that  maps  the  technical  reviews  to  the  acquisition 
process  described  in  DOD  5000  documentation 

■  Aligned  with  DODI  5000.02 

Policy  and  process  described  in  NAVAI  Rl  NST  4355. 19D 

■  Applies  to 

Executive  Officer  (PEO)  programs  involved  with  the  design, 
development,  test  and  evaluation,  acquisition,  in-service  support  and 
disposal  of  naval  aviation  weapon  systems  and  equipment 

Phases 

L  -  ends  with  Milestone  A  (to  begin  technology 

development) 

-  ends  with  Milestone  B  (to  begin  SD&D) 

-  ends  w/  Milestone  C  (LRI P) 

-  ends  with  I  nitial  Operating  Capability  ( I OC) 
(maintenance) 

1  Was  previously  called  Concept  Refinement 

2  Was  previously  called  System  Development  &  Demonstration 


Changes  Based  on  new  5000.02 


- '  -  ^  *  /'* 


B 


IOC 


FOC 


Materiel 

Solution 

Analysis 


Technology 

Development 


Engineering  and 
Manufacturing  Development 


Operations  & 
Support 


Concept  Refinement  to  Materiel  Solution  Analysis 

System  Development  &  Demonstration  back  to 
Engineering  &  Manufacturing  Development 

The  Preliminary  Design  Review  will  be  either  before  or 
after  Milestone  B  as  defined  in  the  program's  acquisition 
strategy 


Two  Pass/ Six  Gate  Process 


Objective:  Establish  a  disciplined  and  integrated 
process  for  requirements  and  acquisition  decision¬ 
making  within  DON,  endorsing  or  approving  key 
J  oint  Capabilities  I  ntegration  and  Development 
Systems  (J  Cl  DS)  and  acquisition  documents  at  Gate 
reviews,  and  facilitating  decisions  regarding 
required  Navy  and  Marine  Corps  capabilities  and 
acquisition  of  corresponding  material  solutions. 

The  SETR  and  the  Two  Pass/ Six  Gate  process  are  complimentary 
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DON  Requirements/ Acquisition  Two-Pass/Six-Cate  Process  with  Development  of  a  System  Design  Specif  ication 

(illustrated  example  for  program  initiation  at  Milestone  A) 


DON  Requirements 


*  DON  CIO  pre-certification,  Investment  Review  Board  certification,  and  Defense  Business  System  (DBS) 
Management  Committee  approval  prior  to  obligation  of  funding  for  a  DBS  program  when  cost  >  $  1  million 
**  Capability  Production  Document  (CPD)  reviews  will  be  chaired  by  CNO/CMC 


AOA 

Analy  si  s  of  Alter nati ve 

IBR 

Integrated  Baseline  Review 

ASN(RD&A) 

Asst  Secretary  of  the  Navy  (Research,  Development  and  Acquisition) 

ICD 

Initial  Capabilities  Document 

p 

CBA 

C  ap  abi  1  iti  es-  Based  As  sessment 

JRGC 

Joint  Requirements  Oversight  Council 

CD 

Concept  Decision 

PEO 

Program  Executive  Officer 

n 

CDD 

Capability  Development  Document 

RFP 

Request  for  Proposal 

i—1 

0 

CMC 

Commandant  of  the  Marine  Corps 

SDD 

System  Development  &  Demonstration 

u 

CNO 

Chief  of  Naval  Operations 

SDS 

System  Design  Specification 

P 

|  -• 

CONORS 

Concept  of  Operations 

SSAC 

Source  Selection  Advisory  Council 

fD 

CSB 

Configuration  Steering  Board 

HQMC 

Headquarters  Marine  Corps 

Major  Phases  of  SETR 
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SETR  Software  Aspects 


■  Software  is  a  key  component  in  almost  every 
SETR 

■  The  NAVAI R  Software  Enterprise  has  developed 
a  set  of  templates  for  each  review 

Defines  the  contents  of  the  software  section  for  the 
review 

I  ndudes  a  list  of  items  to  be  covered  along  with  the 
entrance  criteria  as  it  applies  to  software 
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Example  of  Items  from  SRR  Template 

■  SW  Development  Team 

■  I  ntegrated  Mater  Schedule  Highlighted  with  Software 
Milestones 

■  Software  Entrance  Criteria 

■  Requirements  Analysis  and  Allocation  Methodology 

■  System  Specifications  T ree 

■  Contract  Data  Requirements  List  (CDRL) 

■  Software  Development  Strategy 

■  Software  Development  Process 

■  SW  Safety,  I  nformation  Assurance  and  Security  requirements 

■  Software  Supplier  Management 

■  Software  Measurement 

■  Software  Risk  Assessment  with  Mitigation  Strategies 
I  ssues  and  Concerns 


15 


Systems  Requirements  Review 

(SRR) 
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SRR 

■  Technical  assessment  establishing  the  system  specification  of 
the  system  under  review  to  ensure  a  reasonable  expectation 
of  being  judged  operationally  effective  &  suitable 

■  Ensures  the  Capability  Development  Document  (CDD),  DOD 
Directives,  statutory  and  regulatory  guidance,  and  applicable 
public  law  has  been  correctly  and  completely  represented  in 
the  system  specification  and  can  be  developed  within 
program  cost  and  schedule  constraints 


■  Assesses  the  performance  requirements  as  captured  in  the 
system  specification,  and  ensures  the  preferred  system 
solution  is: 

Consistent  with  the  system  specification 

Correctly  captures  derived  and  correlated  requirements 

Has  well  understood  acceptance  criteria  and  processes 

Is  achievable  through  available  technologies  resulting  from  the  Technology 
Development  phase 
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SRR-I  and  SRR-II 


For  major  programs  going  through  both  concept 
refinement/ materiel  solution  analysis  and  technology 
development,  there  will  be  2  SRRs 

■  SRR-I 

Technical  assessment  establishing  the  specification  of  the 
system  under  review,  previously  represented  by  the 
Performance  Based  Specification  (PBS),  to  continue  the 
requirements  decomposition  process  prior  to  MS  B 

■  SRR-II 

This  review  ensures  the  contractors  participating  in  the 
Technology  Development  (TD)  Phase  understands  that  the 
requirements  of  the  contract  including  the  system  specification, 
SOW,  CDD,  DoD  Directives,  statutory  and  regulatory  guidance, 
and  applicable  public  law  has  been  correctly  and  completely 
represented  in  the  system  specification  and  can  be  developed 
within  program  cost  and  schedule  constraints 
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■  Software  Development  Plan  (SDP) 

Schedule 
Processes 

Software  Development  Environment 
System  Specification 
Initial  Software  Requirements  Description  (SRD) 
Modeling  and  Simulation  Plan 
Supporting  Products 

Systems  Engineering  Plan 
Risk  Management  Plan 
Measurement  Plan 
Quality  Assurance  Plan 
Measurement  Data 
Cost 
Size 

Requirements  Volatility 


Technical  Readiness  Assessment 

(TRA) 
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TRA 


■  A  systematic  metrics- based  process  that 
assesses  the  maturity  of  Critical 
Technology  Elements  (CTEs)  by  an 
independent  panel  of  technical  experts 

■  Applicable  to  all  acquisition  category 
(ACAT)  programs  per  Secretary  of  the 
Navy  Instruction  5000. 2C 

■  May  be  combined  with  a  SRR 
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Definitions 


Technology  Readiness  Level  Definitions 

TRL 

Leve 

1 

Criteria 

1 

Basic  principles  observed  and  reported:  Transition  from  scientific 
research  to  applied  research.  Essential  characteristics  and 
behaviors  of  systems  and  architectures.  Descriptive  tools  are 
mathematical  formulations  or  algorithms. 

2 

Technology  concept  and/or  application  formulated:  Applied 
research.  Theory  and  scientific  principles  are  focused  on  specific 
application  area  to  define  the  concept.  Characteristics 
of  the  application  are  described.  Analytical  tools  are  developed  for 
simulation  or  analysis  of  the  application. 

3 

Analytical  and  experimental  critical  function  and/or 
characteristic  proof-of-concept:  Proof  of  concept  validation. 

Active  Research  and  Development  (R&D)  is  initiated  with  analytical 
and  laboratory  studies.  Demonstration  of  technical  feasibility  using 
breadboard  or  brassboard  implementations  that  are  exercised  with 
representative  data. 

Component/subsystem  validation  in  laboratory  environment: 

4  Standalone  prototyping  implementation  and  test.  Integration  of 
technology  elements.  Experiments  with  full-scale  problems  or  data 

_ sets. _ 

System/subsystem/component  validation  in  relevant 
environment:  Thorough  testing  of  prototyping  in  representative 

5  environment.  Basic  technology  elements  integrated  with  reasonably 
realistic  supporting  elements.  Prototyping  implementations  conform 
to  target  environment  and  interfaces. 


Definitions  cont. 


Technology  Readiness  Level  Definitions  continued 

TRL 

Level 

Criteria 

6 

System/subsystem  model  or  prototyping  demonstration  in  a 
relevant  end-to-end  environment  (ground  or  space):  Prototyping 
implementations  on  full-scale  realistic  problems.  Partially  integrated 
with  existing  systems.  Limited  documentation  available.  Engineering 
feasibility  fully  demonstrated  in  actual  system  application. 

7 

System  prototyping  demonstration  in  an  operational 
environment  (ground  or  space):  System  prototyping 
demonstration  in  operational  environment.  System  is  at  or  near 
scale  of  the  operational  system,  with  most  functions  available  for 
demonstration  and  test.  Well  integrated  with  collateral  and  ancillary 
systems.  Limited  documentation  available. 

8 

Actual  system  completed  and  "mission  qualified"  through  test 
and  demonstration  in  an  operational  environment  (ground  or 

space):  End  of  system  development.  Fully  integrated 
with  operational  hardware  and  software  systems.  Most  user 
documentation,  training  documentation,  and  maintenance 
documentation  completed.  All  functionality  tested  in  simulated  and 
operational  scenarios.  Verification  and  Validation  (V&V)  completed. 

9 

Actual  system  "mission  proven"  through  successful  mission 
operations  (ground  or  space):  Fully  integrated  with  operational 
hardware/software  systems.  Actual  system  has  been  thoroughly 
demonstrated  and  tested  in  its  operational  environment.  All 
documentation  completed.  Successful  operational  experience. 
Sustaining  engineering  support  in  place. 
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I  ntegrated  Baseline  Review 

(IBR) 


24 


IBR 


Employed  by  Program  Managers  ( PMs)  throughout  the 
life  of  projects  where  Earned  Value  Management  (EVM) 
is  required 

■  The  I BR  establishes  a  mutual  understanding  of  the 
project  Performance  Management  Baseline  (PMB)  and 
provides  for  an  agreement  on  a  plan  of  action  to 
evaluate  risks  inherent  in  the  PMB  and  the  management 
processes  that  operate  during  project  execution 

■  Assessment  of  risk  within  the  PMB  and  the  degree  to 
which  the  following  have  been  established: 

Technical  scope  of  work 

Project  schedule  key  milestones 

Resources 

Task 

Rationale 

Management  Processes 
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System  Functional  Review 

(SFR) 
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SFR 

Technical  assessment  establishing  the  system  functional  baseline  of 
the  system  under  review  to  ensure  a  reasonable  expectation  of 
being  judged  operationally  effective  and  suitable 


Functional  requirements  for  operations  and  maintenance  are 
assigned  to  sub-systems,  hardware,  software,  or  support  after 
detailed  reviews  of  the  architecture  in  the  environment  it  will  be 
employed 

The  system's  lower  level  performance  requirements  are  evaluated  to 
determine  whether  they  are  fully  defined  and  consistent  with  the 
mature  system  concept,  and  whether  traceability  of  lower- level  systems 
requirements  to  top-level  system  performance  and  the  CDD  is 
maintained 


Risk  Management,  Measurement,  Quality  Assurance,  & 
Configuration  Management  processes  fully  functional 


SFR  Products 

Updates  to  SRR  products 
SDP 

System  Specification 
Modeling  and  Simulation  Plan 
Supporting  Products 

■  Systems  Engineering  Plan 

■  Risk  Management  Plan 

■  Measurement  Plan 

■  Quality  Assurance  Plan 

■  Cost  and  Size  Estimates 


Updated  (more  detail)  Software  Requirements  Description  (SRD) 
I  nterface  Control  Documents 
Draft  Test  Plan 
Measurement  Data 
Cost 
Size 

Requirements  Volatility 


Software  Specification  Review 

(SSR) 
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Technical  assessment  establishing  the  software  requirements 
baseline  to  ensure  the  preliminary  design  and  ultimately  the 
software  solution  has  a  reasonable  expectation  of  being 
judged  operationally  effective  and  suitable 


A  review  of  the  finalized  Computer  Software  Configuration 
Item  (CSCI)  requirements  and  operational  concept 

Software  Requirements  Specification  (  )  or  Software 

Requirements  Description  (SRD);  Interface  Requirements 
Specif ication(s)  (IRS)  or  Software  Interface  Requirements 
Description  (  );  Software  Integration  Plan;  and  the  user's 

Concept  of  Operation  Description  or  User  Documentation 
Description 


SSR  Products 


Updates  to  SRR  and  SFR  products 

Final  Software  Requirements  Specification  (SwRS)  or  Software 


Requirements  Description  (SRD) 

■  Requirements  Verification  Matrix 

CDD  to  specifications  to  SwRS  or  SRD 

■  Interface  Control  Documents  (I CDs)  and  Interface 
Requirements  Specification  (I  RS)  or  Software  I  nterface 
Requirements  description  (SI  RD) 

■  Declassification,  Anti-Tamper,  Open  Architecture,  and 
I  nformation  Assurance  requirements 

■  Completed  Test  Plan 

■  Software  I  ntegration  Plan 

■  Measurement  Data 

Cost 

Size 

Requirements  Volatility 
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Preliminary  Design  Review 

(PDR) 


32 


PDR 


Technical  assessment  establishing  the  physically  allocated 
baseline  to  ensure  that  the  system  has  a  reasonable 
expectation  of  being  judged  operationally  effective  and 
suitable 

■  captured  in  subsystem 
product  specifications  for  each  configuration  item  in  the 
system  and  ensures  that  each  function,  in  the  functional 
baseline,  has  been  allocated  to  one  or  more  system 
configuration  items 

■  Subsystem  specifications  for  hardware  and  software,  along 
with  associated  Interface  Control  Documents  (I CDs), 
enable  detailed  design  or  procurement  of  subsystems 

■  A  successful  review  is  predicated  on  the  Team's 
determination  that  the 
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PDR  Products 


■  Updates  to  SRR,  SFR,  and  SSR  products 

■  Top  Level  Software  Design  Description  and/or 
Software  Architecture  Description 

■  Completed  Test  Plan 

■  Draft  Test  Procedures 

■  Traceability  from  design  documentation  to 
subsystem  test  requirements 

■  Representative  mission  profiles 

■  Measurement  Data 

Size 

Defects 

Requirements  Volatility 


Critical  Design  Review 

(CDR) 
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CDR 


Technical  assessment  establishing  the  build  baseline  to 
ensure  that  the  system  has  a  reasonable  expectation  of 
being  judged  operationally  effective  and  suitable 

■  as  captured  in  product 
specifications  for  each  configuration  item  in  the  system, 
and  ensures  that  item  has  been  captured  in  detailed 
documentation 

■  Product  specifications  for  software  enable  coding  of  the 
Computer  Software  Configuration  Item  (CSCI ) 

■  A  successful  review  is  predicated  on  the  Team’s 
determination  that  the 
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CDR  Products 


■  Updates  to  SRR,  SFR,  SSR,  &  PDR  products 

All  specifications  and  requirements  documentation  are 
complete/ stable 

■  Detailed  Software  Design  Description 

■  Units  Test  Procedures 

■  Traceability  Verification  Matrix 

CDD  to  specifications  to  requirements  documentation  to  design  to 
test  procedures 

Traceability  in  both  directions 

■  Measurement  Data 

Size 

Defects 

Maturity 

Requirements  Volatility 
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I  ntegration  Readiness  Review 

(IRR) 
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IRR 


Technical  assessment  establishing  the  configuration  to  be 
used  in  integration  test  to  ensure  that  the  system  has  a 
reasonable  expectation  of  being  judged  operationally 
effective  and  suitable 

■  A  product  and  process  assessment  to  ensure  that 


Assess  prior  component  or  unit  level  testing  adequacy,  test 
planning,  test  objectives,  test  methods  and  procedures,  scope  of 
tests,  and  determines  if  required  test  resources  have  been 
properly  identified  and  coordinated  to  support  planned  tests 

Verifies  the  traceability  of  planned  tests  to  program,  engineering 
data,  analysis,  and  certification  requirements 

■  Testing  is  based  upon  the  Test  Plan  (TP) 

Begun  in  requirements  phase  and  completed  during  design 
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■  Updates  to  SRR,  SFR,  SSR,  PDR,  &  CDR 


products 

■  Approved  Integration  Test  Plan 

■  Approved  Integration  Test  Procedures 

■  Format  for  I  ntegration  Test  Report 

■  Completed  Integration  Test  Verification  Matrix 

■  Measurement  Data 


Qua  I  ity/ matu  rity 
Defects 
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Test  Readiness  Review 

(TRR) 
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TRR 


■  Technical  assessment  establishing  the  configuration  used  in  test 
to  ensure  that  the  system  has  a  reasonable  expectation  of  being 
judged  operationally  effective  and  suitable 

■  TRR  is  a  multi-disciplined 


Assesses  prior  unit  level  and  system  integration  testing  adequacy,  test 
planning,  test  objectives,  test  methods  and  procedures,  scope  of  tests,  and 
determines  if  required  test  resources  have  been  properly  identified  and 
coordinated  to  support  planned  tests 

Verifies  the  traceability  of  planned  tests  to  program,  engineering,  analysis, 
and  certification  requirements 

Determines  the  completeness  of  test  procedures  and  their  compliance  with 
test  plans  and  descriptions 

Assesses  the  impact  of  known  discrepancies  to  determine  if  testing  is 
appropriate  prior  to  implementation  of  the  corrective  fix 

■  The 
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■  Updates  to  SRR,  SFR,  SSR,  PDR,  CDR,  and  I RR  products 

■  Traceability  Analysis 

■  Approved  Test  Plan 

■  Approved  Test  Procedures 

■  Format  for  T est  Report 

■  Completed  Test  Verification  Matrix 

■  Software  Version  Document 

■  Measurement  Data 

Qual  ity/ maturity 
Defects 
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Flight  Readiness  Review 

(FRR) 
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FRR 


Technical  assessment  establishing  the  configuration  used  in 
flight  test  to  ensure  that  the  system  has  a  reasonable 
expectation  of  being  judged  operationally  effective  and 
suitable 

with: 

NAVAI R  airworthiness  standards  met 

Objectives  clearly  stated 

Flight  test  data  requirements  clearly  identified 

Acceptable  risk  management  plan  defined  and  approved 

Ensures  that: 

Proper  coordination  has  occurred  between  engineering  and  flight 
test 

All  applicable  disciplines  understand  and  concur  with: 

■  The  scope  of  effort  that  has  been  identified 

■  How  this  effort  will  be  executed  to  derive  the  data  necessary  (to  satisfy 
airworthiness  and  test  and  evaluation  requirements)  to  ensure  the 
weapon  system  evaluated  is  ready  to  proceed  to  flight  test 


FRR  Products 


Updates  to  SRR,  SFR,  SSR, 
TRR  products 


PDR,  CDR,  I RR,  & 


Approved  Flight  Test  Plan 


Approved  Test  Points/ Flight  Scenarios/ Knee 
Cards 


Format  for  Flight  Test  Report 
Software  Version  Document 
Measurement  Data 


Qual  ity/ maturity 
Defects/Test  Hour 


Operational  Test  Readiness  Review 

( OTRR) 
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OTRR 

Multi-disciplined  product  and  process  assessment  to 


■  Successful  performance  during  Operational  Evaluation 
(OPEVAL)  generally  indicates  the  system  being  tested  is 
effective  and  suitable  for  Fleet  introduction 

The  decision  to  enter  production  may  be  based  on  this  successful 
determination 

■  Of  critical  importance  to  this  review  is  the  understanding 
of 

■  Operational  requirements  defined  in  the  CPD  must  match 
the  Requirements  tested  to  in  the  Test  and  Evaluation 
Master  Plan  (TEMP) 
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OTRR  Products 


■  Updates  to  SRR,  SFR,  SSR,  PDR,  CDR,  I RR,  TRR, 
&  FRR  products 

■  Approved  Test  &  Evaluation  Master  Plan  (TEMP) 

■  Approved  Test  Points/ Flight  Scenarios 

■  Format  for  Operational  Test  Report  &  Quick-look 
Report 

■  Measurement  Data 

Qua  I  ity/ matu  rity 
Defects/Test  Hour 
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NAVAI R- Specific  Lessons  Learned 
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Requirements  Lessons  Learned 

Some  developers  tend  to  resist  documenting 
requirements  in  a  requirements  traceability  tool 

I  nability  to  trace  requirements  back  to  customer's/ sponsor's 
requirements 

Requirements  creep  -  adding  requirements  not  needed  to  meet 
user's/ customer's  desires 

Lack  of  concurrence  among  the  stakeholders  of  the 
requirements  (collaboration) 

Key  contributor  to  requirements  instability,  which  leads  to  cost 
and  schedule  problems 

Requirements  too  loose/broadly  written,  making  the 
requirements  decomposition  more  difficult 


Requirements  Lessons  Learned  cont. 


■  Tendency  to  begin  preliminary  design  before 
requirements  done  or  maturity  of  the 
requirements  verified 


■  Resistance  to  having  a  requirements  change 
control  board  early  in  the  requirements  phase 

■  Lack  of  requirements  stability  measure  (metric) 

■  Requirements  phase  too  little  time 
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Design  Lessons  Learned 


Tendency  to  combine  preliminary  design  and  detailed 
design  into  a  single  phase  to  save  time 

Peer  reviews  tend  to  have  more  defects  because  designs  not 
thought  out  well 

Tendency  to  begin  coding/ production  before  design  maturity 
verified 


■  See  some  confusion  between  architecture  definition  and 
design 

These  developers  tend  to  begin  coding  early  and  call  it 
prototyping 
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V  — 


Agile  Programming 
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Coding  Lessons  Learned 


Size  tends  to  go  up  and  amount  of  reuse  tends  to  go  down  as  a 
function  of  time 

Tend  to  underestimate  the  amount  of  effort  required  for  reusing 
code 

Consider  if  new  code  will  be  cheaper  ■ 

Can  be  as  much  as  1.5  times  the  cost  of  new  code  fi 

Auto  code  generator  use  increasing 
Problems/ defects  come  from  humans 


Peer  reviews  are  extremely  important  during  this  phase 
Detects  problems  when  they  are  cheaper  to  fix 
I  ntegrated  unit  testing  tends  to  get  shortened  when  schedules  slide 
Consider  sliding  schedule  or  reducing  content  if  schedule  cannot  slide 

Resource  planning  (labs,  tools,  and  people)  not  necessarily  well 
thought  out,  especially  when  there  is  a  hardware- in- the- loop  lab 
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Testing  Lessons  Learned 

■  Late  development  of  test  plans  and  procedures 

May  not  be  able  to  get  all  the  resources  needed 

May  not  have  the  proper  review  or  secured  commitment  to  the 
testing 

■  Too  little  time  for  testing 

Have  to  rush  through  tests  and  may  overlook  problems 
Have  to  cut  out  some  of  the  tests 

Have  to  test  concurrent  with  development,  which  may  cause 
retest  when  tested  areas  of  the  software  are  changed 

■  Automated  testing  promotes 

Better  coverage  during  testing 
More  efficient  use  of  staff 
Repeatability  of  testing 

Efficient  usage  of  test  facilities  during  2nd  &  3rd  shifts 
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Testing  Lessons  Learned  cont. 

■  Don't  document  test  sessions 

Anomalies  discovered  in  the  test  session  may  get  overlooked 

Don't  have  data  to  show  what  has  been  tested  to  support 
decisions 

■  Readiness  for  next  step 

■  Flight  clearances 

■  Production  decisions 

Cant  take  credit  for  the  testing  performed  and  may  have  to  redo 
testing 

■  Wrong  configuration  tested 

■  Contention  for  test  facilities  due  to  poor  up-front 
planning 

Schedule  delays 
Removal  of  tests 
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I  ASSUME  nV  HIGH 
LEVEL  OF  EFFICIENCY 
WILL  BE  RECOGNIZED 
AND  REWARDED. 


I  FOUND  A  CLEVER 
WAY  TO  WRITE  MY 
APPLICATION  CODE 
IN  ONE  HOUR! 


LET  HE  KNOW  HOW 
THAT  WORKS  OUT 
FOR  YOU, 


FROM  NOW  ON,  I 
EXPECT  YOU  TO 
FINISH  ALL  OF  YOUR 
PROJECTS  IN  ONE 
HOUR. 


YOU 

COULD 

HAVE 

WARNED 

HE. 


OTHERWISE  I'LL 
ASSUME  YOU'RE 
RIPPING  OFF  THE 
COMPANY 


YOU  D  ID 
ALL  OF 
THAT  IN 
ONE  HOUR? 


THAT'S 
NOT  HO  W 
EXPERIENCE 
WORKS, 


Scott  Adams,  Inc./DlSt.  by  UFS,  Inc. 


s,  I  incorporated  and  is  provided  for  non-commercial  use  by  United  Features  Syndicate,  I  ncorporated. 


Schedule  Compliance 


Schedule  I  ssues 

■  Beginning  tasks  before  maturity  or  readiness  has 
been  verified  rarely  if  ever  saves  time  and 
funding 

JSF 

■  The  J  SF  started 

/ 

,  and 


FCS 

■  The  current  FCS  practice  is  to 

with  software  developers  reporting 

that 

,  compromising  the  amount  of  time 

allotted  for  testing 
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Schedule  Driven  Demos 


v* 


i  the  technology  DEMO  | 

the  software 

]  ISN'T  1002  / 

INCOMPLETE .  j 

□ 

v - 

|o|  fifl 

IF  IT  HAD  A  USER 
INTERFACE  YOU 
WOULD  SEE  SOME¬ 
THING  HERE... 
HERE.  .  .AND  SO  MI¬ 
STIMES  HERE.  / 


c 

f 


r  AND  THEN  YOU'D 

BE  SAYING,  *X 
GOTTA  GET  ME~ 
SOME  OF  that: 
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Schedule  Lessons  Learned 


■  Race  through  requirements 

■  Produce  a  superficial  design 

■  Rush  into  coding 

■  Software  tends  to  be  delivered  later  than  if  it  were  developed  with  a  rational 
plan 

Lack  of  staffing  allocation  across  the  schedule 

■  Most  common  problem  is  staff  brought  in  too  late 

Overlapping  of  phases  (to  save  time)  and  beginning  next  phase  with  an 
immature  product 

Critical  path  is  not  evident 

■  I  mpacts  risk  management 

■  Puts  you  in  reaction  mode  (chaotic)  because  you  did  nothing  to  mitigate 

I  nsufficient  detail  to  assess  risks 

Unknown  linkages  and  interdependencies  of  the  tasks  identified 
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Software  Suppliers  Lessons 

I  n  an  article  in  CrossTalk  Magazine  on  software  acquisition  in  the 
Army,  Edgar  Dalrymple,  the  Program  Manager  for  the  Future 
Combat  Systems  Brigade  Combat  Team  and  Associate  Director  of 
Software  and  Distributed  Systems,  when  answering  a  question 
making  one  change  in  the  way  the  government  procures  software: 
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Subcontractor  Management  Lessons 


■  I  n  major  contracts  where  the  prime  has 
subcontracted  out  subsystems,  we  have  seen: 

Software  requirements  regarding  process  maturity 
and  development  of  deliverables  not  passed  down  to 
the  subcontractor 

Prime  subcontractor  management  personnel  not 
experienced  in  software  development 

■  Results  in  latency  of  requirements  changes  to  software 

Rework 

Schedule  delay 

■  Requirements  volatility 
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Summary 


■  The  Systems  Engineering  Technical  Review 
process  is  an  iterative  program  timeline  that 
maps  to  DODI  5000.02  acquiaition  timeline  and 
is  compatible  with: 

Basic  Systems  Engineering 

CMMI 

I EEE  12207 

■  There  are  many  software  development  lessons 
that  NAVAI R  has  learned  from  its  participation  in 
SETR  events 
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